feat(power): validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 - #2437
feat(power): validate multinode GPU energy metrics / 校验多机 GPU 能耗指标#2437edwingao28 wants to merge 5 commits into
Conversation
中文:新增多机 srt-slurm 电力 artifact 消费模块:严格镜像 producer 合约做独立复算(不信任 stored publication_valid),校验 producer SHA pin、拓扑、样本身份与 window 绑定,梯形积分输出 role 级与全局能耗指标。
中文:多机分支接入 aggregate_power_multinode 消费钩子(best-effort 镜像单机路径,REQUIRE_POWER=1 才失败),internal-error 回退同步清理 role 指标键。
中文:benchmark-multinode-tmpl 新增 require-power/power-producer-sha 输入、REQUIRE_POWER/POWER_PRODUCER_SHA 环境与 power audit bundle 上传;e2e-tests 仅向 multi-node 泳道透传;test-process-result 接入多机模块与测试。
|
Claude finished @edwingao28's task in 4m 43s —— View job Review: multinode GPU power consumption
Summary: One 🟡 warning posted inline — when integration overflows to |
|
Validated in CI on real hardware, strict mode (REQUIRE_POWER=1):
|
中文:溢出的 per-GPU 能量此前以 Infinity 字面量写入 sidecar,严格 RFC 8259 解析器(jq/JS/Go)会拒收;改为 null 并在溢出测试中断言 sidecar 严格可解析。
…r3-multinode-consumption
Stacked on #2323 — base is
wenyao/validated-single-node-power. After #2323 merges this will rebase and retarget tomain. Only the last 3 commits are new.What this does
Consumes the srt-slurm
dcgm-powerartifact package (v1 wire contract:power/{manifest.json, samples.csv, windows/*.json}) for disaggregated multinode runs, re-validates it independently, and patches energy metrics into the per-concurrency agg JSON:power_valid,avg_power_w,avg_total_gpu_power_w,total_gpu_energy_j,joules_per_successful_query,joules_per_input_token,joules_per_output_token,joules_per_total_token)prefill_gpu_energy_j,decode_gpu_energy_j,prefill_joules_per_input_token,decode_joules_per_output_token. Role energy is board-level energy over the full serving window. It is not kernel-phase energy.Trust model: the producer's stored
publication_validis never trusted. The consumer recomputes validity from the persisted bytes, mirroring srt-slurm's own offline validator (identity sextet, strict sample parsing, per-device UUID 1:1, timestamp monotonicity, window status invariants, fixed 3.0 s max sample gap, window bracketing, stored-evidence equality). It then cross-checks the stored verdict; any disagreement isproducer_verdict_mismatchandpower_valid=0. The producer git commit must equal thePOWER_PRODUCER_SHApin in every mode. The workspace result is bound to its measurement window by canonical-JSON SHA256 against the original atLOGS/<result_path>— the launcher renames results when copying, so names can't be compared.Default mode is best-effort: invalid telemetry publishes
power_valid=0plus a validation sidecar and never fails the benchmark.REQUIRE_POWER=1fails the job after the audit artifacts exist.Validated against a real GB200 artifact
Canary run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30587402515 (Qwen3.5-397B FP8, 1P1D TP4, concurrency 4, producer pin
6609d46a4c74ed66fc2f7014d7e790efe1d23bde).publication_valid: Truerecomputed.power_valid=1, avg 343.72 W per GPU (2749.74 W across 8),total_gpu_energy_j=189898.14over the 69.06 s window, prefill 73214.34 J + decode 116683.80 J (sums exactly).samples.csvagrees to 2e-10 relative error.het_group=None. The topology gate now enforces het-group distinctness only when het groups are present. Het-job deployments (e.g. GB300) keep the strict check.Known deltas (documented, left unchanged)
_load_benchmark_data(feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323) bases its duration tolerance on wall time; the producer uses the reported duration. The divergence is reject-side only: a ~1% clock-skew band on >50 s runs where the consumer withholds metrics the producer would accept. Not touched here to keep feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323's reviewed code intact.run()shares the raise-before-write pattern this PR closes in the multinode path (non-finite metrics now flip the verdict to invalid instead of dying mid-patch). The single-node path is untouched.Tests
utils/test_aggregate_power_multinode.py— 22 tests. A self-consistent package builder, then every contract gate tampered one at a time; each rejection asserted in both modes.utils/test_process_result.py::TestMultinodePower— 5 wiring tests through the real subprocess path..github/workflows/test-process-result.ymlnow collects the new suite and triggers on the new files.中文:
堆叠在 #2323 上——base 为
wenyao/validated-single-node-power,#2323 合并后会 rebase 并改回 main,只需 review 最后 3 个 commit。本 PR 消费 srt-slurm 多机
dcgm-powerartifact(v1 wire contract),向 agg JSON 写入 8 个整机指标和 4 个 role 指标。prefill/decode 能耗是整个服务窗口的板级能耗,不是算子阶段能耗。信任模型:不信任 producer 存储的
publication_valid。消费端从持久化字节独立复算(与 srt-slurm 离线校验器同一套检查),再与 stored verdict 交叉校验,不一致即producer_verdict_mismatch。producer SHA 必须命中POWER_PRODUCER_SHApin。workspace 结果与 window 通过 canonical JSON SHA256 绑定(launcher 拷贝时会改名,不能比文件名)。默认 best-effort,无效遥测只记power_valid=0不失败;REQUIRE_POWER=1才严格失败。已用真实 GB200 canary artifact 离线验证(数据见上),并借此在合并前修掉一个真实 bug:GB200 1P1D 是普通两节点 Slurm 作业,所有设备
het_group=None,拓扑门限现在只在 het group 存在时强制互异(GB300 het 作业仍严格)。单元测试 22 个(逐一破坏每个 contract gate,双模式断言)+ process_result 接线测试 5 个;test-process-result workflow 已接入新文件。